Learning to Prompt for Vision-Language Models

Large pre-trained vision-language models like CLIP have shown great potential in learning representations that are transferable across a wide range of downstream tasks. Different from the traditional representation is based mostly on discretized labels, pre-training aligns images and texts common feature space, which allows zero-shot transfer to task via prompting, i.e., classification weights synthesized natural language describing classes interest. In this work, we show major challenge for deploying such practice prompt engineering, requires domain expertise extremely time-consuming—one needs spend significant amount time words tuning since slight change wording could huge impact performance. Inspired by recent advances research processing (NLP), propose Context Optimization (CoOp), simple approach specifically adapting CLIP-like image recognition. Concretely, CoOp prompt’s context with learnable vectors while entire parameters kept fixed. To handle different recognition tasks, provide two implementations CoOp: unified class-specific context. Through extensive experiments 11 datasets, demonstrate as few one or shots beat hand-crafted prompts decent margin able gain improvements over engineering more shots, e.g., 16 average around 15% (with highest reaching 45%). Despite being learning-based approach, achieves superb generalization performance compared model using prompts.